Efficient distributional reinforcement learning with Kullback-Leibler divergence regularization

In this article, we address the issues of stability and data-efficiency in reinforcement learning (RL). A novel RL approach, Kullback-Leibler divergence-regularized distributional (KL-C51) is proposed to integrate advantages both (KL) one framework. KL-C51 derived Bellman equation TD errors regularized by KL divergence a perspective explored approximated strategies properly mapping corresponding Boltzmann softmax term into distributions. Evaluated not only several benchmark tasks with different complexity from OpenAI Gym but also six Atari 2600 games Arcade Learning Environment, method clearly illustrates positive effect regularization including exclusive exploration behaviors smooth value function update, demonstrates an improvement compared other related baseline approaches.